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EXECUTIVE  SUMMARY 

The  basic  concept  behind  the  Cooperative  Operations  in  Urban  Terrain  (COUNTER)  program 
was  one  of  layered  sensing  involving  a  small  Unmanned  Air  System  (UAS)  over  an  urban  area 
that  was  capable  of  dispensing  smaller  Micro  UAS’s  to  fly  at  lower  altitudes  and  take  closer 
looks  at  partially  obscured  targets.  This  paper  focuses  on  the  Vigilant  Spirit  Control  Station  user 
interface  that  was  enhanced  to  allow  one  operator  to  simultaneously  control  multiple  UAS’s 
during  COUNTER  simulations  and  flight  tests.  The  use  of  NATO  STANAG  4586  for  vehicle 
connectivity  is  discussed  as  a  way  to  provide  a  common  interface  to  heterogeneous  UAS’s.  An 
overview  of  the  primary  interface  components  is  provided  including  the  use  of  a  video  mosaic 
and  digital  video  recorder  (DVR)  capability  to  aid  an  operator  in  detecting  targets  while 
interacting  with  multiple  simultaneous  video  feeds.  Finally,  a  detailed  description  is  provided  of 
the  user  interface  developed  to  interact  with  COUNTER’S  cooperative  control  algorithms.  These 
algorithms  provided  a  dynamic  mission  planning  capability  that  automatically  allocated  vehicles 
to  targets  and  generated  mission  flight  plans  to  enable  the  vehicles  to  fly  to  the  targets  in  an 
optimized  and  cooperative  manner. 


1.0  INTRODUCTION 

The  basic  concept  for  the  Cooperative  Operations  in  Urban  Terrain  (COUNTER)  program 
was  one  of  layered  sensing  involving  a  small  Unmanned  Air  System  (UAS)  over  an  urban  area 
that  was  capable  of  dispensing  smaller  Micro  UAS’s  to  fly  at  lower  altitudes  and  take  closer 
looks  at  partially  obscured  targets.  A  typical  mission  involved  one  MLB,  Inc.  Bat-III  aircraft  and 
one  or  more  Applied  Research  Associates  (ARA),  Inc.  Nighthawk  aircraft.  The  Bat-III 
contained  a  single  gimbaled  camera  that  had  limited  operator  slew  and  zoom  capability.  Each 
Nighthawk  had  a  fixed  forward  and  side  looking  camera  that  the  operator  could  switch  between 
as  desired. 

The  focus  of  the  COUNTER  research,  from  the  Air  Force  Research  Laboratory  (AFRL)  Air 
Vehicles  Directorate  perspective,  was  the  development  and  subsequent  testing  of  the  dynamic 
mission  planning  (DMP)  components  that  generated  mission  plans  enabling  multiple  vehicles  to 
simultaneously  and  efficiently  survey  the  targets  requested  by  the  operator.  In  order  to 
effectively  test  these  targets  requested  by  the  operator.  In  order  to  effectively  test  these 
algorithms,  COUNTER  needed  an  operator  interface  that  enabled  an  operator  to  choose  targets, 
generate  vehicle  plans,  and  manage  the  overall  mission  of  the  vehicles.  This  requirement  was 
perfectly  suited  for  a  teaming  arrangement  with  the  Human  Effectiveness  Directorate,  Warfighter 
Interface  Division,  System  Control  Interface  Branch’s  Vigilant  Spirit  (VS)  program. 


‘VS  Program  Manager,  AFRL/RHCI,  2210  8th  St.  Bldg  146  Rm.  122,  Area  B,  WPAFB,  OH,  45433 
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VS  teamed  with  COUNTER  from  the  early  stages  utilizing  the  Vigilant  Spirit  Control 
Station  (VSCS).  VSCS  is  a  multi-UAS  control  station  capability  that  enables  one  operator  to 
simultaneously  control  multiple  vehicles.  The  focus  of  the  research  from  the  Human 
Effectiveness  Directorate  perspective  was  the  operator  interface  components  and  underlying 
software  architecture  required  to  allow  one  operator  to  control  multiple  vehicles.  VSCS  was 
enhanced  to  meet  the  unique  requirements  of  COUNTER  and  provided  the  interface  through 
which  all  simulations  and  flight  testing  was  completed.  VSCS  became  known  as  the  “Face  of 
COUNTER”  since  operators  were  able  to  see  the  capabilities  of  COUNTER  come  to  life  via 
VSCS. 


2.0  VIGILANT  SPIRIT  BACKGROUND 

To  fully  understand  and  appreciate  the  capabilities  VS  brought  to  COUNTER,  a  short 
overview  of  VS’s  history  is  in  order.  The  roots  of  what  is  currently  the  VS  program  go  back 
several  years  to  a  research  program  simply  called  Operator  Vehicle  Interface  (OVI) The  focus 
of  this  somewhat  generically  named  program  was  the  development  of  interface  concepts  that 
allowed  one  operator  to  simultaneously  control  four  lethal  unmanned  air  vehicles  (UAV’s)  in  the 
Suppression  of  Enemy  Air  Defenses  mission.  Figure  1  -  OVI  shows  a  sample  view  of  the 
interface. 
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Figure  1  -  OVI 


During  the  course  of  OVI,  engineers  became  members  of  a  UCAV  cockpit  working  group. 
Concepts  were  developed  and  tested  that  allowed  a  single  operator  to  manage  several  vehicles 
simultaneously;  commanding  them  to  perform  tasks  such  as  capturing  Synthetic  Aperture  Radar 
(SAR)  imagery  and  engaging  targets. 
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The  OVI  follow-on  program  became  what  is  today  known  as  Vigilant  Spirit  with  its  primary 
product  being  the  Vigilant  Spirit  Control  Station.  The  lessons  learned  from  various  UCAV 
related  projects  enabled  the  VS  team  to  establish  numerous  collaborations  across  a  broad  range 
of  vehicles  and  mission  concepts.  These  collaborations,  many  of  which  are  still  ongoing  at  the 
time  of  this  publication,  have  proven  to  be  one  of  the  most  valuable  aspects  in  the  development 
of  VSCS  because  of  the  opportunity  to  explore  the  operator  interface  requirements  across  the 
various  domains.  For  example,  in  support  of  the  Long  Range  Strike  program,  the  VS  team 
enhanced  VSCS  to  support  a  conceptual  unmanned  high  speed  bomber.  The  vehicle  contained  a 
large  mixed  weapons  load  and  interacted  with  a  DMP  capability.  For  the  Automated  Aerial 
Refueling  (AAR)  program2,  the  VSCS  gained  the  addition  of  formats  to  enable  an  operator  to 
manage  the  refueling  of  a  flight  of  four  UCAV’s.  The  process  involves  the  flight  rendezvous 
with  the  tanker,  the  join-up  with  the  tanker,  and  finally  the  actual  refueling  of  each  vehicle  much 
like  the  current  refueling  procedures  of  a  flight  of  four  manned  fighters.  Numerous  other 
projects  have  utilized  VSCS  in  various  capacities. 

Although  the  types  of  vehicles  and  missions  planned  for  COUNTER  were  at  the  opposite 
end  of  the  spectrum  from  a  long  range  strike  vehicle,  the  flexibility  demonstrated  by  VSCS 
through  the  various  collaborations  indicated  it  would  be  well  suited  for  COUNTER.  The  next 
several  sections  will  describe  in  more  detail  the  capabilities  added  to  VSCS  to  support 
COUNTER.  These  additions  were  made  while  maintaining  a  focus  on  commonality  across  the 
various  vehicles  and  missions  that  VSCS  has  been  involved. 


3.0  VIGILANT  SPIRIT  COMPONENTS  OVERVIEW 

The  overall  VSCS  system  utilized  for  COUNTER  is  comprised  of  several  key  components 
as  illustrated  in  Figure  2.  This  section  of  the  paper  provides  a  brief  overview  of  these 
components  with  respect  to  COUNTER  simulation  and  flight  testing.  A  more  detailed  operator 
interface  discussion  is  provided  in  the  next  section. 

3.1  Vigilant  Spirit  Software  Architecture 

The  VS  software  architecture  has  been  designed  to  support  a  wide  variety  of  uses.  It  needs  to 
be  able  to  support  a  myriad  of  mission  scenarios  and  vehicle  platforms.  In  addition  it  must  be 
able  to  facilitate  a  multitude  of  research  mechanisms  ranging  from  completely  virtual,  basic 
human  factors  studies,  as  well  as  real-life  flight  tests  with  all  the  features  of  a  fielded  system.  To 
efficiently  author  software  that  meets  these  goals,  a  highly  flexible  software  architecture  has 
been  developed. 
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Figure  2  -  Vigilant  Spirit  Components 


The  VS  suite  of  software  has  been  written  entirely  using  commercial  off-the-shelf  (COTS) 
software  development  tools.  Emphasis  has  been  on  designing  applications  to  be  run  on  PCs 
running  the  Microsoft  Windows  operating  system.  The  bulk  of  the  VS  software  is  written  in  the 
C#  programming  language.  This  language  choice  provides  many  benefits  such  as  the  existence  of 
various  rapid  application  development  tools,  memory  management,  and  binary  compatibility 
with  other  languages  such  as  Visual  Basic  .NET  and  C++/CLI.  A  minority  of  the  VS  software  is 
written  in  the  native  C++  programming  language  with  various  calls  into  the  ubiquitously  used 
OpenGL  graphics  language.  An  attempt  is  always  made  to  develop  with  the  latest  compilers  and 
development  tools  and  to  achieve  as  much  standards-compliance  as  possible. 

Given  the  wide  variety  of  setups  the  VSCS  must  support,  it  is  critical  that  experiment- 
specific  code  modifications  are  kept  to  a  minimum.  For  this  reason  the  VS  architecture  is  highly 
data  driven.  A  large  number  of  data  files,  used  to  hold  various  types  of  configuration  data,  are 
utilized  by  the  VSCS  and  its  supporting  software  components.  Some  of  the  types  of  data  include 
initial  conditions,  vehicle  characteristics  and  capabilities,  sensor  descriptions,  network  setup, 
output  paths,  control  station  display  layouts,  and  symbol  sets.  In  keeping  with  conformance  to 
cutting  edge  commercial  standards,  these  data  files  are  represented  almost  exclusively  using  the 
extensible  markup  language  (XML). 

Depending  on  the  type  of  mission  being  executed,  the  mission  operator  may  need  any 
number  of  tools  at  his  or  her  disposal  to  execute  the  mission  successfully.  For  the  VSCS,  these 
tools  are  provided  in  the  form  of  a  suite  of  graphical  user  interface  (GUI)  components,  each 
providing  the  operator  with  the  ability  to  perform  a  specific  task.  From  the  software  developer’s 
perspective,  these  tools  are  each  authored  independently.  Using  the  VSCS’s  underlying  data 
files,  the  particular  subset  of  these  tools  that  is  loaded  for  a  particular  mission  is  completely 
configurable. 
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A  robust  “plug-in”  architecture  has  been  developed  within  the  VS  software  architecture  to 
support  the  configurable  loading  of  its  various  GUI  components.  These  components  can  be 
authored  in  a  self-contained  manner  so  that  providing  additional,  mission-specific  functionality  is 
easy  to  achieve  with  minimal  development  time.  Additionally,  there  are  various  customization 
points  within  VSCS  and  its  supporting  components  for  modules  that  do  not  provide  a  user 
interface  but  provide  some  mission-specific  functionality  (such  as  scripting,  audio  cues,  payload 
modeling,  etc). 

The  underlying  framework  upon  which  the  VS  software  is  written  has  been  designed  for  a 
high  level  of  reusability.  Using  this  basic  software  engineering  principle,  developing  reusable 
software  enhances  the  software’s  flexibility  by  shortening  the  development  time  required  to 
create  new  modules.  Many  fundamental  software  engineering  principles  are  used  including  the 
utilization  of  some  common  design  patterns.  A  model-view-controller  (MVC)  design  pattern  is 
commonly  used  within  the  framework  to  separate  the  underlying  data,  the  user  interface 
components,  and  the  logic  required  to  perform  a  particular  task  or  respond  to  a  particular  event. 
This  allows  identical  functionality  to  be  displayed  to  the  user  in  a  variety  of  ways  with  little  code 
duplication. 

3.2  Vehicle  Connectivity 

Figure  3  shows  the  COUNTER  connectivity  diagram  utilized  for  flight  testing.  The  VSCS 
has  been  designed  to  scale  across  a  multitude  of  vehicle  platforms.  It  is  expected  to  support 
missions  in  both  real-life  and  simulation.  These  variables  make  the  task  of  making  the  control 
station  as  interoperable  as  possible  very  important.  The  main  area  where  interoperability  is 
implemented  in  regard  to  the  VSCS  is  its  data  link  interface  (DLI).  In  particular,  the  DLI 
between  the  control  station  and  the  vehicles  it  is  controlling.  VSCS  utilizes  the  NATO 
STANAG  4586  protocol  as  its  DLI. 

The  aims  of  STANAG  4586  and  the  interoperability  aims  of  the  VSCS  join  very  nicely.  The 
standard  introduces  two  major  software  components:  the  Core  UAV  Control  System  (CUCS) 
and  the  Vehicle  Specific  Module  (VSM).  The  CUCS  encompasses  the  human  computer  interface 
and  in  this  circumstance  is  the  VSCS.  The  VSM  is  a  software  module  that  contains  the  logic 
needed  to  translate  the  4586  DLI  messages  into  messages  that  the  vehicle  understands.  The  VSM 
is  responsible  for  translating  commands  coming  from  the  CUCS  to  the  vehicle  and  translating 
reports  from  the  vehicle  to  the  CUCS.  It  is  important  to  note  that  the  VSM  is  not  the  vehicle 
itself  but  a  bridge  connecting  the  CUCS  to  the  vehicle.  The  bridge  design  is  one  that  is  familiar 
in  the  software  engineering  discipline  and  is  frequently  used  to  achieve  interoperability. 
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Figure  3  -  COUNTER  Connectivity  Diagram 


The  4586  standard  describes  various  components  that  are  used  to  implement  an 
interoperable  UAV  control  system.  The  largest  segment  of  the  standard  is  the  definition  of  what 
is  referred  to  as  the  Common  Message  Format  (CMF)  and  the  messages  that  follow  this  format, 
making  up  the  data  link  interface.  The  messages  described  in  the  standard  provide  the  data 
required  for  the  implementation  of  a  multitude  of  vehicle  commands  ranging  from  navigation  to 
weapons  delivery  and  sensor  interaction.  The  format  describes  what  fields  each  message  has  and 
how  to  interpret  those  fields. 

For  the  COUNTER  flight  tests,  both  vehicle  manufacturers  created  VSM’s  that  acted  as 
bridges  converting  VSCS  4586  messages  to  each  vehicle’s  native  message  set.  This  enabled 
VSCS  to  communicate  with  the  COUNTER  vehicles  in  the  same  manner  it  has  for  other  vehicle 
integrations.  The  COUNTER  VSM’s  were  tested  using  the  Vigilant  Spirit  4586  CMF  Toolkit. 
This  toolkit  was  developed  by  the  VS  team  to  aid  with  the  process  of  integrating  with  a  new 
VSM.  It  consists  of  a  code  generator  that  generates  the  DLI  CMF  messages  in  C#,  Java,  and 
C++.  The  code  generator  makes  it  very  easy  to  extend  the  message  set  with  any  required  custom 
messages.  To  aid  in  the  debugging  process  during  this  recurring  phase  of  development,  a  utility 
program  was  written  named  the  Vigilant  Spirit  4586  CMF  Analyzer.  The  program  was 
implemented  to  function  similarly  to  a  network  packet  analyzer  but  with  a  focus  on 
receiving/viewing  and  creating/sending  CMF  messages. 

3.3  Dynamic  Mission  Planning  (DMP) 

In  the  context  of  the  VSCS,  DMP  is  a  generic  term  used  to  refer  to  a  capability  to  alter  a 
vehicle’s  planned  course  of  action  in  flight.  The  VSCS  focus  is  not  on  building  DMP 
capabilities,  but  rather  on  how  to  allow  an  operator  to  efficiently  interact  with  them  to 
accomplish  mission  requirements.  The  VSCS  architecture  treats  a  DMP  as  a  separate  process  or 
service  that  VSCS  interacts  with.  If  desired,  each  different  vehicle  type  could  have  its  own 
unique  DMP  capability  tuned  specifically  to  its  specifications.  The  location  of  the  DMP  is  also 
flexible.  In  some  cases  it  may  be  desirable  to  have  the  DMP  on  the  vehicle.  In  other  cases,  due 
to  processing  limitations  of  the  vehicle  or  communications  constraints,  it  may  be  more 
advantageous  to  place  the  DMP  with  the  control  station. 
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For  COUNTER,  the  DMP  capability  consisted  of  the  Cooperative  Control  Algorithm  (CCA) 
planner,  Search  Pattern  Planner  (SPP),  and  Stochastic  Controller  (SC)3.  These  capabilities  were 
bundled  together  in  a  separate  process  that  ran  on  the  same  computer  as  the  VSCS.  The  VSCS 
communicated  with  this  process  using  a  set  of  custom  messages  that  were  built  to  satisfy  unique 
DMP  requirements.  These  messages  were  built  using  the  VS  4586  CMF  Toolkit  following  the 
same  DLI  format  as  the  messages  used  to  communicate  with  the  vehicles.  The  details  of  the 
COUNTER  DMP  are  beyond  the  scope  of  this  paper. 

3.4  Simulation 

To  effectively  test  most  systems,  portions  of  the  environment  they  operate  in  need  to  be 
simulated.  The  VSCS  development  for  COUNTER  was  no  different.  The  overall  COUNTER 
project  consisted  of  three  levels  of  simulation  including  the  Engineering  Simulation,  the 
Operational  Simulation,  and  what  we  are  discussing  here  as  the  Developmental  Simulation.  This 
simulation  capability  strives  to  enable  the  VSCS  engineers  to  conduct  their  testing  as  if  they 
were  in  the  field  with  real  aircraft.  Almost  without  exception,  the  VSCS  engineers  were  not 
required  to  build  any  special  capabilities  to  do  lab  testing  versus  flight  test.  For  flight  tests,  the 
VSCS  lab  systems  were  disconnected  from  power,  network,  and  video,  taken  to  the  field,  re¬ 
connected  and  were  ready  to  go.  This  required  our  vehicle  simulations  to  communicate  in  the 
same  manner  as  flight  test  vehicles.  The  vehicles’  cameras  along  with  the  virtual  worlds  they 
were  viewing  needed  to  be  simulated.  While  simulation  development  is  not  the  primary  focus  of 
the  VS  program,  it  has  been  a  necessity  for  effective  development  and  testing  of  the  VSCS.  The 
VSCS  has  been  architected  to  utilize  not  only  VS’s  limited  simulation  capability,  but  also  the 
more  sophisticated  capabilities  of  other  organizations. 

4.0  VSCS  USER  INTERFACE  PRIMARY  COMPONENTS  OVERVIEW 

As  the  title  of  the  paper  indicates  and  was  mentioned  in  the  introduction,  “VSCS,  The  Face 
of  COUNTER”  has  really  been  a  good  catch  phrase  to  illustrate  the  VSCS’s  involvement  with 
various  projects.  The  phrase  was  coined  by  former  COUNTER  program  manager  Captain  Nidal 
Jodeh  (AFRL/RBCA).  The  point  was  made  that  much  of  COUNTER’S  primary  research  interest 
was  in  the  behind  the  scenes  cooperative  control  algorithms,  but  how  the  operator  interacted  with 
them  was  brought  to  life  through  what  you  saw  with  via  the  innovative  user  interfaces  developed 
for  the  VSCS.  This  analogy  has  proven  true  for  VSCS  development  across  numerous  projects 
that  the  VSCS  has  been  applied  to  and  tends  to  be  true  for  any  interface  development  work. 
From  the  user  interface  perspective,  COUNTER  presented  the  VSCS  team  with  several  new 
challenges  that  will  be  highlighted  in  the  next  several  sections. 

The  VSCS  operator  interface  incorporates  a  flexible  modular  design  that  can  be  configured 
to  accommodate  several  diverse  mission  requirements.  Individual  graphical  user  interface 
components,  or  tools,  can  be  included  as  desired  for  a  particular  mission.  The  VSCS’s  tool 
arrangement  can  be  configured  to  accommodate  any  particular  computer  system  display 
capability  and  resolution.  Figure  4  shows  a  typical  VSCS  two  display  configuration  (Each  24” 
diagonal  with  a  1920  x  1200  pixel  resolution)  and  tool  layout  used  during  COUNTER  flight 
tests.  This  example  illustrates  a  three  vehicle  configuration  involving  one  Bat-III  and  two 
Nighthawk  vehicles.  The  primary  tools  utilized  include  the  Summary  Panel,  Tactical  Situation 
Display  (TSD),  Video  Mosaic  Tool,  and  the  Video  Tool.  The  VSCS  contains  other  tools,  but 
only  the  ones  used  most  frequently  for  COUNTER  will  be  discussed. 
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Figure  4  -  YSCS  Typical  COUNTER  Configuration 


4.1  Summary  Panel 

Along  the  left  edge  of  the  left  display  is  the  Summary  Panel.  The  Summary  Panel  is 
designed  to  provide  a  quick  look  summary  of  important  information  for  each  aircraft  tailored  to 
its  current  mission  phase.  An  individual  pane  is  provided  for  each  aircraft  and  a  sample  is  shown 
in  Figure  5.  To  help  the  operator  quickly  distinguish  between  different  aircraft,  four  unique 
features  are  utilized.  First  is  the  icon  shape  in  the  upper  left  comer  representing  the  vehicle  (Bat- 
III),  second  is  a  unique  color  (Orange),  third  is  the  call  sign  (BAT- A),  and  fourth  is  a  unique 
digit  representing  the  number  of  aircraft  we’re  controlling  (1).  These  features  are  consistently 
used  across  the  various  VSCS  tools  to  distinguish  between  vehicles. 


Figure  5  -  Summary  Panel 


4.2  Tactical  Situation  Display  (TSD) 

The  next  major  tool  available  to  the  operator  was  referred  to  as  the  Tactical  Situation 
Display  (TSD).  The  TSD  consists  of  underlay  maps  or  imagery  of  an  operational  area  with 
information  overlays.  The  underlay  surface  supports  standard  aviation  charts  and  geo  registered 
imagery.  An  icon  of  each  vehicle  is  drawn  on  the  TSD  at  its  proper  geographical  location  and 
orientation.  Other  information  such  as  the  current  vehicle  plan,  preview  plans,  sensor  footprints, 
and  target  locations  are  also  shown.  The  TSD  is  the  primary  situational  display  used  by  the 
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operators  to  monitor  and  provide  status  information  for  each  vehicle.  It  can  be  thought  of  as  any 
standard  mapping  application  but  is  tailored  to  facilitate  vehicle  control.  Various  options  are 
available  for  easily  slewing  the  map  around  and  changing  the  map  scale.  Other  options  are  also 
available  for  turning  on  or  off  a  choice  of  overlay  items. 

The  bottom  area  of  the  TSD  contains  a  pop-up  component  called  the  command  tool.  This  is 
the  area  where  the  operator  can  control  various  aspects  of  the  mission  and  interact  with  the  DMP 
to  create  new  mission  plans.  The  component  can  be  shown  or  hidden  as  desired  by  the  operator 
but  was  left  up  most  of  the  time  during  COUNTER  testing.  The  primary  command  tool  tabs  or 
pages  utilized  for  COUNTER  were  Nav,  Loiter,  Plans,  DMP,  DMP  Search,  and  Camera. 

4.3  Video  Mosaic  Tool 

Located  to  the  right  of  the  TSD,  as  shown  in  Figure  4,  is  the  Video  Mosaic  Tool.  This  is 
one  of  the  user  controls  that  was  first  created  and  tested  in  support  of  COUNTER.  This 
component  was  created  using  a  COTS  software  development  kit  (SDK)  and  supported  only  one 
video  source  at  any  particular  time.  The  tool  essentially  painted  the  operator  a  larger  view  using 
multiple  video  frames.  A  good  analogy  is  taking  a  deck  of  cards  and  fanning  them  out  on  a  table 
with  each  card  being  a  different  video  frame.  This  tool  provided  the  operator  additional 
situational  awareness  and  helped  to  provide  a  more  stabilized  view  of  targets  during  turbulent 
conditions. 

4.4  Video  Tool 

The  configuration  as  seen  in  Figure  4  shows  three  separate  Video  Tools  stacked  along  the 
right  edge  of  the  right  display.  Each  Video  Tool  allows  the  operator  to  select  an  analog  video 
source  being  transmitted  from  one  of  the  vehicles  currently  under  VSCS  control.  Figure  6  allows 
a  closer  look  at  one  of  the  video  tools.  The  control  station  also  provided  a  Digital  Video 
Recorder  (DVR)  capability  that  allowed  the  operator  to  scroll  back  in  the  video  feeds  if  desired. 
This  capability  proved  useful  when  flying  low  over  targets  with  the  Nighthawk.  The 
Nighthawks  were  typically  flown  around  22-24  knots  which  is  not  very  fast,  but  at  125  feet  off 
the  ground,  positively  identifying  targets  could  be  challenging  since  they  were  in  view  for  such  a 
short  time.  With  turbulence  and  video  distortions  the  task  became  even  tougher.  The  DVR 
allowed  the  operator  to  slide  the  video  back  in  time  and  freeze  frame  on  the  target  to  allow  a 
little  more  time  for  target  identification.  The  Video  Tool  DVR  contains  other  functionality 
similar  to  that  found  on  home  DVR  systems.  Features  such  as  pause,  fast  forward,  etc.  exist  and 
are  useful  to  the  target  acquisition  process,  but  will  not  be  discussed  here. 
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5.0  INTERACTING  WITH  COUNTER  ALGORITHMS 

As  was  mentioned  in  the  introduction  to  this  report,  the  basic  concept  for  the  COUNTER 
program  was  one  of  layered  sensing  involving  a  small  UAS  over  an  urban  area  that  was  capable 
of  dispensing  smaller  micro  UAS’s  to  take  closer  looks  at  partially  obscured  targets4.  This 
section  provides  a  few  examples  of  an  operator  using  the  VSCS  to  interact  with  the  COUNTER 
algorithms.  These  examples  are  typical  of  what  was  done  during  flight  testing  to  execute  a 
COUNTER  test  card.  The  VSCS  also  provides  several  convenient  methods  to  allow  the  operator 
to  easily  perform  tasks  such  as  configuring  loiter  parameters  and  controlling  payloads  but  the 
discussion  of  these  items  is  beyond  the  scope  of  this  paper. 

For  the  example  illustrated  here,  we  will  assume  the  operator  is  in  control  of  a  single  Bat- 
III  with  a  call  sign  BAT-A  and  two  Nighthawks  with  call  signs  MAV-A  and  MAV-B.  Each  of 
the  vehicles  are  in  predefined  circular  loiters  around  different  points  with  an  overall  VSCS 
configuration  as  illustrated  in  Figure  4.  The  operator’s  task  is  to  pick  targets  and  utilize  the 
COUNTER  DMP  capabilities  to  generate  plans  that  Nighthawks  will  fly  at  low  level  to  view  the 
targets. 

Figure  7  shows  a  TSD  with  the  Command  Tool  area  open  along  the  bottom  edge.  The 
Command  Tool’s  DMP  tab  is  colored  blue  indicating  that  the  DMP  Command  Tool  is  the  one 
currently  selected.  With  the  DMP  Command  Tool  selected,  the  next  task  the  operator  needs  to  do 
is  determine  which  of  his  available  vehicles  to  use.  The  vehicle  selector  control  shows  that  both 
Nighthawks  have  been  selected.  This  is  indicated  by  the  background  of  the  selector  button  being 
filled  with  the  color  assigned  to  the  vehicle.  Just  below  the  vehicle  selector  is  the  payload 
options  area.  This  shows  the  list  of  available  payload  items  that  can  be  used  for  generating  a 
plan.  In  this  case  the  only  item  available  is  a  camera  that  is  shown  as  selected  by  the  blue 
background  color.  Once  the  vehicles  are  selected,  each  one  needs  to  be  configured  for  the 
desired  altitude  and  speed  settings.  This  is  accomplished  by  selecting  the  Vehicle  Settings 
option  and  entering  the  proper  values. 
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Now  that  the  vehicles  are  selected  and  configured,  the  operator  has  three  options  for 
entering  target  points.  The  first  way  is  to  monitor  one  of  the  vehicle’s  video  feeds  and  click  on  a 
desired  location  directly  in  the  Video  Tool.  This  causes  a  target  entry  to  be  added  to  the  target 
list  as  well  as  a  geo-referenced  target  point  to  be  added  on  the  TSD.  The  operator’s  next  option 
is  to  select  a  target  point  directly  from  the  TSD.  This  is  accomplished  by  holding  down  the  Shift 
key,  moving  the  cursor  to  the  desired  location  over  the  TSD  and  clicking  the  left  mouse  button. 
The  final  way  is  to  enter  a  point  directly  by  typing  in  the  coordinates  using  the  controls  on  the 
DMP  Command  Tool. 

Figure  7  shows  six  Electro-Optical  (EO)  or  camera  targets  in  the  target  list  that  are  labeled 
as  EO-5  to  EO-IO.  These  target  points  are  also  shown  on  the  TSD  as  grey  outlined  squares  with 
the  same  labels.  The  operator  now  has  the  option  to  constrain  the  approach  angle  to  a  target.  In 
this  example,  the  operator  selects  target  EO-6  and  in  the  constraints  area  specifies  that  the 
vehicle  should  approach  the  target  from  a  heading  of  090  degrees  with  a  standoff  distance  of 
1000  feet.  The  enabled  constraint  is  the  one  with  the  blue  On  button  selected  to  the  right  of  the 


Figure  7  -  Target  Selections 


constraint  values.  Up  to  four  constraints  can  be  added  to  a  target.  At  this  point  the  operator 
selects  the  green  Request  New  button  to  request  that  plans  be  generated  for  each  of  the  selected 
aircraft  using  the  targets  and  constraints  entered. 

Figure  8  shows  the  TSD  with  a  proposed  plan  generated  by  the  COUNTER  algorithms  for 
both  MAV-A  and  MAV-B.  The  proposed  plans  are  represented  as  dashed  lines  using  each 
vehicle’s  assigned  color.  The  lines  are  dashed  indicating  that  they  are  only  proposed  plans  that 
have  not  yet  been  sent  to  the  vehicle.  The  DMP  Command  Tool  shows  the  estimated  duration  of 
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each  of  the  plans  along  with  the  number  of  waypoints  each  contains.  The  operator  is  given  the 
option  to  reject  or  accept  both  the  plans.  Note  that  the  algorithm  provided  the  vehicle  to  target 
allocation  as  well  as  the  optimized  plans  to  visit  each  target.  For  this  example,  the  operator 
accepts  the  plans  which  cause  them  to  be  sent  to  each  of  their  respective  vehicles.  Since  the 
vehicle’s  navigation  mode  is  set  to  loiter  (LTR),  they  do  not  begin  flying  the  plans  right  away.  It 
is  not  shown,  but  the  plans  show  up  as  dim  solid  lines  on  the  TSD  once  the  vehicles 
acknowledge  receipt  of  the  plans.  The  operator  selects  the  NAV  Command  Tool  tab  and  places 
both  Nighthawk  vehicles  into  waypoint  (WPT)  mode.  At  this  point  each  of  the  vehicle  plans 
show  up  as  brighter  solid  lines  indicating  that  they  are  the  active  plans  currently  being  flown. 


Figure  8  -  Pending  Plans 
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Figure  9  -  Nighthawks  Flying  Plans 


Figure  9  now  shows  the  TSD  zoomed  in  with  both  Nighthawks  well  into  each  of  their 
missions.  The  TSD  shows  that  the  sensor  footprint  of  the  Nighthawk  with  call  sign  MAV-A  is 
now  over  one  of  its  targets  at  the  same  time  the  Bat-III  is  looking  at  the  same  target  from  above. 
Now  that  the  Nighthawks  are  flying  their  low  level  missions,  the  operator  has  several  tools  at  his 
disposal  to  aid  in  seeing  the  selected  targets  in  the  Nighthawk’s  video.  Even  though  the 
Nighthawks  travel  at  a  slow  speed,  identifying  a  specific  target  can  still  be  a  challenging  task 
since  the  sensor  footprint  is  so  small  and  the  vehicle  travels  over  the  target  area  very  quickly  at 
low  levels.  The  VSCS  has  several  built  in  tools  to  aid  with  the  target  identification.  The  first 
being  the  DVR  capability  mentioned  in  an  earlier  section.  Figure  10  shows  a  view  of  the  TSD 
with  the  vehicle  of  interest  shown  at  its  current  location  along  with  what  is  called  a  ghost  ship 
view.  The  ghost  ship  is  shown  with  a  dashed  sensor  footprint  outline  and  represents  where  the 
vehicle  was  at  some  past  time.  The  ghost  ship  is  created  whenever  the  DVR  capability  is  used  to 
view  past  video.  This  correlation  can  be  useful  when  trying  to  find  the  section  of  video  when  the 
vehicle  was  over  one  of  the  targets. 
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Figure  10  -  Ghost  Ship 


6.0  CONCLUSION 

Any  successful  program  draws  upon  the  talents  of  research  individuals  across  a  multitude  of 
disciplines.  The  combination  of  the  AFRL  Air  Vehicle  Directorate  and  the  Human  Effectiveness 
Directorate  proved  to  be  a  successful  integration  of  both  control  law  algorithms  and  human- 
centered  UAS  supervisory  control.  The  need  for  a  common  UAS  ground  control  station  that 
could  be  utilized  in  a  research  and  flight  test  environment  helped  to  mature  the  VSCS  over  the 
past  several  years.  The  VSCS  became  an  integral  test  bed  utilized  across  a  multitude  of 
programs  and  basic  research  organizations.  The  VS  team  has  developed  a  mature  and  robust 
software  architecture  using  state-of-the-art  development  tools  and  has  incorporated  COTS  to 
help  reduce  costs  and  enhance  functionality. 

A  key  component  for  making  a  successful  simulation  environment  has  been  the  use  of  the 
NATO  STANAG  4586  message  set.  The  VS  team  recognized  the  importance  for  implementing 
a  common  standard  messaging  protocol  to  effectively  communicate  to  a  diverse  set  of  UAS’s. 
The  development  of  the  VS  STANAG  4586  Common  Message  Format  Toolkit  has  been  integral 
to  the  success  of  allowing  a  wide  variety  of  vehicle  vendors  to  create  VSM’s.  This  has  allowed 
our  simulation  environment  to  scale  across  vehicle  platforms  and  provide  both  a  simulation  and 
flight-test  ready  system. 

The  COUNTER  program  has  been  an  integral  facilitator  of  the  many  features  that  have 
enhanced  the  situational  awareness  of  the  operator-vehicle  dynamic.  Being  the  “Face  of 
COUNTER”,  the  VSCS  enabled  the  AFRL  Air  Vehicles  Directorate  cooperative  control 
algorithms  to  be  tested  in  simulation  as  well  as  live  flight  tests.  This  DMP  capability  has 
allowed  for  further  research  to  be  done  evaluating  the  effectiveness  of  multiple  vehicle  control  in 
an  urban  environment.  Effective  sensor  management  techniques  through  the  introduction  of  the 
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DVR  capability  and  live  video  mosaicing,  has  proven  to  enhance  the  operator  awareness  through 
the  use  of  cooperative  vehicle  planning.  Further  research  will  be  conducted  looking  into  adding 
additional  features  to  the  CCA  algorithms  such  as  terrain  avoidance,  weather,  airspace 
management,  road  network  planning,  and  many  others.  Search  patterns  and  stochastic 
controllers  will  also  be  evaluated  to  aid  the  operator  in  making  decisions  during  the  DMP 
process  and  target  identification. 
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LIST  OF  ACRONYMS 

AAR 

Automated  Aerial  Refueling 

ARA 

Applied  Research  Associates 

AFRL 

Air  Force  Research  Laboratory 

CCA 

Cooperative  Control  Algorithm 

CMF 

Common  Message  Format 

COTS 

Commercial  Off-The-Shelf 

COUNTER 

Cooperative  Operations  in  Urban  TERrain 

cues 

Core  UAV  Control  Station 

DLI 

Data  Link  Interface 

DMP 

Dynamic  Mission  Planning 

DVR 

Digital  Video  Recorder 

EO 

Electro-Optical 

GUI 

Graphical  User  Interface 

MVC 

Model  View  Controller 

OVI 

Operator  Vehicle  Interface 

SAR 

Synthetic  Aperture  Radar 

SC 

Stochastic  Controller 

SDK 

Software  Development  Kit 

SPP 

Search  Pattern  Planner 

TSD 

Tactical  Situation  Display 

UAS 

Unmanned  Air  System 

UAV 

Unmanned  Air  Vehicle 

UCAV 

Unmanned  Combat  Air  Vehicle 

VS 

Vigilant  Spirit 

VSCS 

Vigilant  Spirit  Control  Station 

VSM 

Vehicle  Specific  Module 

XML 

Extensible  Markup  Language 

16 

Distribution  A:  Approved  for  public  release;  distribution  unlimited. 
88  ABW  Cleared  07/25/2008;  88ABW-2008-4887. 


